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PROFESSOR HOXBY: I'd like to go back to 
July 2004 and tell you about my outlook on charter 
schools at that time. At that time, there were more 
than 3,000 charter schools in the United States, and 
they didn't enroll very many students in the U.S.- 
only about 1.5 percent of the students in the U.S. 
But they were important schools. We researchers 
thought we would learn a lot from them, and edu- 
cational innovators from all parts of the political 
spectrum were involved in making charter schools 
work across the United States. 

Although most charter schools are in urban 
areas like New York City and although most dis- 
proportionately enroll low-income and minority 
students, there are also charter schools in rural ar- 
eas. For instance, there are a good many charter 
schools in Alaska, which mainly enroll Native 
Americans. There are also a number of charter 
schools in rural Texas, rural Minnesota, and rural 
California that cater mainly to the children of farm 
workers. 

In short, in July 2004, there was a diverse set 
of charter schools. D espite that diversity, there was 
tremendous consensus in the research communi- 
ty about how to evaluate charter schools. The con- 
sensus was that we needed to have randomized 
studies, which are easy to do with charter schools 
because most of them have more applicants than 
they have places. 



In order to do a randomized study, we start with 
a charter school's entire pool of apphcants. When 
the school holds a lottery among its apphcants, some 
will randomly be admitted (or "lotteried in") and 
consequently attend charter schools. Some will ran- 
domly not be admitted (the "lotteried- out") and 
consequently continue to attend regular public 
schools. Researchers follow both groups of stu- 
dents, the lotteried- in and lotteried- out, over sever- 
al subsequent years. N early all researchers agree that 
this type of study is the gold standard because ran- 
domization ensures that you're comparing apples 
with apples. We compare students who go to char- 
ter schools to students who wanted to go to charter 
schools. The two groups are not only the same in 
terms of family background, they also are equally 
motivated, doing equally well in regular public 
school, and so on. 

The U.S. D epartment of E ducation has frmded 
several studies that use randomization, and they will 
produce findings in a couple of years. I myself am 
leading some studies that use randomization, and, 
with my co-author Jonah Rockoff of Columbia 
University, released the first rando miz ation-based 
study in May 2004. Let me give you a sense of what 
such a study looks like. We studied the largest charter 
school system in Illinois, which covers many of the 
charter school students in Chicago. The charter 
schools in question are about 80 percent black. 
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Most of the rest of their students are Hispanic. 
About 82 percent of their students come from such 
low- income families that they would qualify for free 
or reduced-price lunch. AU of the students come 
from inner-city neighborhoods. 

We were able to ascertain that the Chicago 
charter school lotteries were indeed fair. The lot- 
teried-in and lotteried-out students were almost 
identical in terms of race, ethnicity, family income, 
home location, special education status, limited 
English proficiency, and prior achievement in the 
regular public schools. 

After following both groups of students, we 
found that the lotteried-in students who attended 
charter schools had math and reading achievement 
that was about 6 percentile points higher than lot- 

If we extrapolate the gains to an entire primary and secondary 
education, a student could gain almost four grade equivalents by 
bang in thecharto: schools as opposed to the regular public schools. 

teried-out students who continued in the regular 
pubhc schools. These gains were achieved after only 
a couple of years. If we extrapolate the gains to an 
entire primary and secondary education, a student 
could gain almost four grade equivalents by being 
in the charter schools as opposed to the regular 
pubhc schools. Ultimately, we won't need to ex- 
trapolate because we'll continue to follow the stu- 
dents until they graduate from high school. Studies 
like ours, which use the randomized method and 
follow children over a long period of time, are un- 
doubtedly the best way to do research on charter 
schools. Moreover, such studies are also a good 
way for the charter schools to learn about them- 
selves. We researchers communicate with the 
schools so that they learn where they're succeed- 
ing and where they're not— math versus reading, 
grades one through three versus grades four 
through six, and so on. 

1 'm hoping that we researchers are able to con- 
tinue doing studies of this type, but the future can 
be potentially undermined by articles like the one 
that appeared on the front page of the N ew York 
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T imes in August of this year. It was one of the most 
irresponsible examples of education reporting that 
1 have ever seen. 

The N ew York T imes gave uncritical promotion 
to a "study" conductedby the American Federation 
of Teachers (AFT) that compared students in char- 
ter schools in the United States with students in 
regular public schools, using the National Assess- 
ment of Educational Progress test. That test is the 
only exam given to a representative sample of stu- 
dents across the entire United States. It is a very 
good, useful exam for plotting national trends and 
for following states in a general way. 

However, the National Assessment of Educa- 
tional Progress is administered to only 3 percent of 
American students and is designed to be represen- 
tative of the entire student population of the U.S. 
This means that it is not appropriate for the use 
that the AFT put it to. Charter- school students are 
only 1.5 percent of students in the United States, 
and the test covers only 3 percent of students. Take 
3 percent of 1.5 percent, and you reahze that the 
charter school results touted by the AFT depended 
on a mere 0.045 percent of the students in the 
United States. That is, the results depending on 
about four one-hundredths of one percent of stu- 
dents in America. 

That is an incredibly small number of students 
on whom to rely. To give you a sense, four one- 
hundreds of one percent of students in New York 
is just thirty- two students. In other words, we're talk- 
ing about relying on the equivalent of one class- 
room of students to evaluate charter schools for 
the entire state of New York. The situation is even 
worse for smaller states, where the results may de- 
pend on fewer than a dozen students! The N ation- 
al Assessment of Educational Progress is just not 
appropriate for evaluating charter schools. 

Even worse, the AFT study compared apples 
to oranges. It compared the average charter school 
student with the average regular public school 
student. That's not a fair comparison because most 
charter school students are economically 
disadvantaged students from inter-city or rural areas. 
They're disproportionately hkely to be minorities. 
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In contrast, the typical regular public school student 
in the United States is a suburban student who is 
not a racial or ethnic minority and who comes from 
a family with average income. 

Not only did the N ew York Times not criticize 
the AFT for comparing apples to oranges, the N eiw 
York Times did not even point out that when the 
AFT compared black students to black students, or 
Hispanic students to Hispanic students, charter 
school and regular pubhc school students performed 
equally well. The N ew York T imes article was not only 
a very irresponsible piece of reporting; it was de- 
liberately misleading. 

My initial response to the AFT study was that 
it was such a lousy study that it did not deserve a 
response. It did not deserve to be considered for 
even a few minutes. Serious education scholars 
all over the United States agreed. Their unani- 
mous response was that the AFT study was poor 
research and highly unscientific. Let me give you 
just one piece of evidence about the unanimity 
of the response. Several people have asked me 
to debate another scholar about the AFT study. 
I 've replied each time by saying that, if they could 
find me a scholar who will support the study. I'd 
be glad to debate him or her. D espite scouring 
the scholarly world, they've not been able to come 
up with a single serious researcher who will de- 
fend the AFT report. 

In America, we need to have education reform 
based on science if we're going to make progress. 
Science is the standard we set for medical research. 
Why should we accept quackery for education? 

Unfortunately, the AFT study received a lot of 
attention, probably owing to the N ew Y ork T imes's 
irresponsible coverage. The attention eventually 
convinced me that it would be a good idea for some- 
one to produce evidence that addressed the AFT 
study's most egregious faihngs. 

Although charter schools are fee-based and 
more autonomous than regular pubhc schools, they 
are public schools. Therefore, in every state in the 
United States, charter school students take exactly 
the same exams, under the same conditions, as 
regular public school students take. Charter school 



students are assessed against the same proficiency 
levels as are regular public school students. 

Therefore, there is no reason to rely on a mi- 
nuscule sample of students. Y ou can get test scores 
for all students in charter schools in the United 
States, and that's exactly what I did. 

I obtained data on each elementary charter 
school in the United States and matched each one 
to a sensible comparison school— the nearest regu- 
lar pubhc school or the nearest regular pubhc school 
with a similar racial composition. These schools 
(which are the same one 93 percent of the time) 
make a good, sensible comparison. Why? Think 
about where a charter school student who is, say, a 
fourth- grader would be if he or she weren't in the 
charter school? He or she would probably be in the 
nearest regular pubhc school. Put another way, what 
decision do the parents of a disadvantaged, elemen- 
tary school-aged child really get to make? They de- 
cide whether to send their child to the local regular 
pubhc school or a nearby charter school. They don't 
get to consider elite suburban schools or schools 
that are twenty-five miles away. 
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F urthffmore, in states that have stronger charter school laws 
and that have had charter schools oparatinglonga:, theprofidaicy 
gains are largo:. 



In short, what I did was compare charter 
schools with the nearest regular public schools, 
which shared not only a neighborhood, but also were 
very similar on race, ethnicity, family income, par- 
ent's education, and so on. When I did this, I found 
that, on average, charter school students are about 
5 percent more likely to be proficient in reading and 
3 percent more likely to be proficient in math. 

These are not dramatic improvements in pro- 
ficiency, but they are significant. Furthermore, in 
states that have stronger charter school laws and 
that have had charter schools operating longer, 
the proficiency gains are larger. Here are some 
examples: 

In Arizona, the state with the largest share of 
students enrolled in charter schools, charter school 
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students are about 10 percent more likely to be pro- 
ficient in reading and math. In Cakfomia, Massa- 
chusetts, and Pennsylvania, charter school students 
are 8 to 9 percent more likely to be proficient in 
reading. In Alaska, Colorado, the District of Co- 
lumbia, New Jersey, Hawaii, Illinois, Louisiana, New 
Jersey, and 0 regon, charter school students are at 
least 12 percent more hkely to be proficient in read- 
ing. North Carolina is the only state in which the 
charter school students were statistically hkely to 
be performing worse than regular pubhc school stu- 
dents in both reading and math. There, they were 
about 4 percent less likely to be proficient. 

There's nothing like a oombination of an inspiration and a threat 
to wake up a manageneit team and that indudes themanageneit 
teams at regular public schools. 

That's a picture of the nation's charter school 
students as we have it now. Y ou'U notice that 1 didn't 
mention New York. Why? New York, like many 
other states, has too few charter school students 
enrolled at this point for us to figure out what's 
happening. 1 laud Chancellor Klein's efforts to im- 
prove charter school availability in N ew York City, 
but the fact is that the state of New York has only 
0.33 percent (one third of one percent) of its stu- 
dents enrolled in charter schools. With such a smah 
share of students, it's too early to draw conclusions. 

0 f course, the goal of education reform is not 
to have a sea of mediocre public schools in which 
there are a few high-performing charter schools. 
What we'd like to see in the long run is high perfor- 
mance at both charter schools and regular public 
schools. A charter school can exhibit pedagogical 
and management innovations, but it can also be a 
threat. There's nothing hke a combination of an 
inspiration and a threat to wake up a management 
team and that includes the management teams at 
regular public schools. Thus, while we'd hke charter 
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schools to innovate and set performance standards, 
we'd also hke to see all regular pubhc schools rise to 
the challenge and match them. 

There is evidence that regular pubhc schools 
are already rising to the challenge in Michigan and 
Arizona, where charter schools represent a signifi- 
cant share of enrohment. In those states, we see 
that regular pubhc schools improve when they face 
a neighboring charter school that can take, say, 6 
percent of local students. We want ah American 
schools to experience similar chahenges and respond 
similarly. 1 cannot emphasize enough that the goal 
is to improve ah schools, not just to have good char- 
ter schools in the United States. 

Improving education in America could not be 
more important. I'm an economist by training and 
1 worry a great deal about the skihs of the country's 
population. We live in a global environment in which 
we cannot protect American workers forever from 
s kill ed workers overseas. 1 wish we could, but that 
is not in the cards. 

Therefore, we need all students in the U.S. to 
become skilled enough to compete in the global 
economy. It's not acceptable to have students who 
are uns kill ed because they are minorities, or are dis- 
advantaged, or just go to lousy schools. Education 
reform is all about ensuring that every child has the 
opportunity to succeed. 

E ducation reform in the United States also ought 
to be based on science. We do not accept drugs that 
do not go through randomized trials, and we cer- 
tainly don't pubhsh quack studies about drugs on the 
front page of the N ew York T imes. Returning to my 
first theme, 1 want to reiterate that randomized stud- 
ies are the best way to understand charter schools. 
We should have the patience to wait for randomized 
studies to produce results. Charter schools are doing 
weh enough - better on average than the regular pubhc 
schools with which they compete— to make us wait 
and find out what works and why. 



4 



> 



Notes 



I 

> 



U) 

H 

H 

C 

H 

m 



5 



I- 

LU 



Notes 



D 

CQ 

U 

> 

u 



6 



> 



Notes 



I 

> 



U) 

H 

H 

C 

H 

m 



For additional information, contact the Manhattan Institute, 52 Vanderbilt Avenue, New York, NY 10017 
Telephone 212.599.7000 • Fax 212.599.3494 • E-mail holsen(a)manhattan-institute.org • www.manhattan-institute.org 
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